Bayesian estimation for the random moderation model: effect size, coverage, power of test, and type І error.

The random moderation model (RMM) was developed based on a two-level regression model to cope with heteroscedasticity in moderation analysis, and normal-distributed-based maximum likelihood (NML) estimation was developed to estimate the RMM. To present an alternative to the NML, this article discusses the effectiveness of Bayesian estimation for the RMM, aiming to explore a more practical method using the popular software Mplus. Through a simulation study, the RMM based on Bayesian estimation was investigated and compared to maximum likelihood (ML) estimations, including the NML and the default ML estimation in Mplus. The results indicated that the Bayesian approach outperformed the two ML estimations. It showed (a) higher accuracy for estimation of the effect size of the moderation effect; (b) higher 95% credibility interval coverage of the true value of the moderation effect; and (c) well-controlled and more stable type I error rates, while powers comparable to the ML estimations were provided.

Joint modeling of action sequences and action time in computer-based interactive tasks.

Process data refers to data recorded in computer-based assessments that reflect the problem-solving processes of participants and provide greater insight into how they solve problems. Action time, namely the amount of time required to complete a state transition, is also included in such data along with actions. In this study, an action-level joint model of action sequences and action time is proposed, in which the sequential response model (SRM) is used as the measurement model for action sequences, and a new log-normal action time model is proposed as the measurement model for action time. The proposed model can be regarded as an extension of the SRM by incorporating action time within the joint-hierarchical modeling framework and as an extension of the conventional item-level joint models in process data analysis. Results of the empirical and simulation studies demonstrated that the model setup was justified, model parameters could be interpreted, parameter estimates were accurate, and taking into account participants' action time further was beneficial for obtaining a deep understanding of participants' behavioral patterns. Overall, the proposed action-level joint model provides an innovative modeling framework for analyzing process data in computer-based assessments from the latent variable modeling perspective.

Longitudinal joint modeling for assessing parallel interactive development of latent ability and processing speed using responses and response times.

To measure the parallel interactive development of latent ability and processing speed using longitudinal item response accuracy (RA) and longitudinal response time (RT) data, we proposed three longitudinal joint modeling approaches from the structural equation modeling perspective, namely unstructured-covariance-matrix-based longitudinal joint modeling, latent growth curve-based longitudinal joint modeling, and autoregressive cross-lagged longitudinal joint modeling. The proposed modeling approaches can not only provide the developmental trajectories of latent ability and processing speed individually, but also exploit the relationship between the change in latent ability and processing speed through the across-time relationships of these two constructs. The results of two empirical studies indicate that (1) all three models are practically applicable and have highly consistent conclusions in terms of the changes in ability and speed in the analysis of the same data set, and (2) additional analysis of the RT data and acquisition of individual processing speed measurements can reveal the parallel interactive development phenomena that are difficult to detect using RA data alone. Furthermore, the results of our simulation study demonstrate that the proposed Bayesian Markov chain Monte Carlo estimation algorithm can ensure accurate model parameter recovery for all three proposed longitudinal joint models. Finally, the implications of our findings are discussed from the research and practice perspectives.

Deterministic Input, Noisy Mixed Modeling for Identifying Coexisting Condensation Rules in Cognitive Diagnostic Assessments.

In cognitive diagnosis models, the condensation rule describes the logical relationship between the required attributes and the item response, reflecting an explicit assumption about respondents' cognitive processes to solve problems. Multiple condensation rules may apply to an item simultaneously, indicating that respondents should use multiple cognitive processes with different weights to identify the correct response. Coexisting condensation rules reflect the complexity of cognitive processes utilized in problem solving and the fact that respondents' cognitive processes in determining item responses may be inconsistent with the expert-designed condensation rule. This study evaluated the proposed deterministic input with a noisy mixed (DINMix) model to identify coexisting condensation rules and provide feedback for item revision to increase the validity of the measurement of cognitive processes. Two simulation studies were conducted to evaluate the psychometric properties of the proposed model. The simulation results indicate that the DINMix model can adaptively and accurately identify coexisting condensation rules, existing either simultaneously in an item or separately in multiple items. An empirical example was also analyzed to illustrate the applicability and advantages of the proposed model.

Don't worry about the anchor-item setting in longitudinal learning diagnostic assessments.

Previous longitudinal assessment experiences for multidimensional continuous latent constructs suggested that the set of anchor items should be proportionally representative of the total test forms in content and statistical characteristics and that they should be loaded on every domain in multidimensional tests. In such cases, the set of items containing the unit Q-matrix, which is the smallest unit representing the whole test, seems to be the natural choice for anchor items. Two simulation studies were conducted to verify the applicability of these existing insights to longitudinal learning diagnostic assessments (LDAs). The results mainly indicated that there is no effect on the classification accuracy regardless of the unit Q-matrix in the anchor items, and even not including the anchor items has no impact on the classification accuracy. The findings of this brief study may ease practitioners' worries regarding anchor-item settings in the practice application of longitudinal LDAs.

Bridging Models of Biometric and Psychometric Assessment: A Three-Way Joint Modeling Approach of Item Responses, Response Times, and Gaze Fixation Counts.

Recently, joint models of item response data and response times have been proposed to better assess and understand test takers' learning processes. This article demonstrates how biometric information such as gaze fixation counts obtained from an eye-tracking machine can be integrated into the measurement model. The proposed joint modeling framework accommodates the relations among a test taker's latent ability, working speed and test engagement level via a person-side variance-covariance structure, while simultaneously permitting the modeling of item difficulty, time-intensity, and the engagement intensity through an item-side variance-covariance structure. A Bayesian estimation scheme is used to fit the proposed model to data. Posterior predictive model checking based on three discrepancy measures corresponding to various model components are introduced to assess model-data fit. Findings from a Monte Carlo simulation and results from analyzing experimental data demonstrate the utility of the model.

DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method.

Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents' problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents' problem-solving competence. The proposed method cannot only estimate respondents' problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents' problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents' problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.

Variable Speed Across Dimensions of Ability in the Joint Model for Responses and Response Times.

Working speed as a latent variable reflects a respondent's efficiency to apply a specific skill, or a piece of knowledge to solve a problem. In this study, the common assumption of many response time models is relaxed in which respondents work with a constant speed across all test items. It is more likely that respondents work with different speed levels across items, in specific when these items measure different dimensions of ability in a multidimensional test. Multiple speed factors are used to model the speed process by allowing speed to vary across different domains of ability. A joint model for multidimensional abilities and multifactor speed is proposed. Real response time data are analyzed with an exploratory factor analysis as an example to uncover the complex structure of working speed. The feasibility of the proposed model is examined using simulation data. An empirical example with responses and response times is presented to illustrate the proposed model's applicability and rationality.

A Markov Estimation Strategy for Longitudinal Learning Diagnosis: Providing Timely Diagnostic Feedback.

Timely diagnostic feedback is helpful for students and teachers, enabling them to adjust their learning and teaching plans according to a current diagnosis. Motivated by the practical concern that the simultaneity estimation strategy currently adopted by longitudinal learning diagnosis models does not provide timely diagnostic feedback, this study proposes a new Markov estimation strategy, which follows the Markov property. A simulation study was conducted to explore and compare the performance of four estimation strategies: the simultaneity, the Markov, the anchor-item, and the separated estimation strategies. The results show that their performance was highly consistent, and they presented in the following relative order: simultaneity > Markov > anchor-item ≥ separated. Overall, although accuracy in parameter estimation is sacrificed slightly with the proposed strategy, it can provide timely diagnostic feedback to practitioners, which is in line with the concept of "assessment for learning" and the needs of formative assessment.

The Development of an Instrument for Longitudinal Learning Diagnosis of Rational Number Operations Based on Parallel Tests.

The precondition of the measurement of longitudinal learning is a high-quality instrument for longitudinal learning diagnosis. This study developed an instrument for longitudinal learning diagnosis of rational number operations. In order to provide a reference for practitioners to develop the instrument for longitudinal learning diagnosis, the development process was presented step by step. The development process contains three main phases, the Q-matrix construction and item development, the preliminary/pilot test for item quality monitoring, and the formal test for test quality control. The results of this study indicate that (a) both the overall quality of the tests and the quality of each item are good enough and that (b) the three tests meet the requirements of parallel tests, which can be used as an instrument for longitudinal learning diagnosis to track students' learning.

The Impact of Sample Attrition on Longitudinal Learning Diagnosis: A Prolog.

Missing data are hard to avoid, or even inevitable, in longitudinal learning diagnosis and other longitudinal studies. Sample attrition is one of the most common missing patterns in practice, which refers to students dropping out before the end of the study and not returning. This brief research aims to examine the impact of a common type of sample attrition, namely, individual-level random attrition, on longitudinal learning diagnosis through a simulation study. The results indicate that (1) the recovery of all model parameters decreases with the increase of attrition rate; (2) comparatively speaking, the attrition rate has the greatest influence on diagnostic accuracy, and the least influence on general ability; and (3) a sufficient number of items is one of the necessary conditions to counteract the negative impact of sample attrition.

A Sequential Higher Order Latent Structural Model for Hierarchical Attributes in Cognitive Diagnostic Assessments.

The higher-order structure and attribute hierarchical structure are two popular approaches to defining the latent attribute space in cognitive diagnosis models. However, to our knowledge, it is still impossible to integrate them to accommodate the higher-order latent trait and hierarchical attributes simultaneously. To address this issue, this article proposed a sequential higher-order latent structural model (LSM) by incorporating various hierarchical structures into a higher-order latent structure. The feasibility of the proposed higher-order LSM was examined using simulated data. Results indicated that, in conjunction with the deterministic-inputs, noisy "and" gate model, the sequential higher-order LSM produced considerable improvement in person classification accuracy compared with the conventional higher-order LSM, when a certain attribute hierarchy existed. An empirical example was presented as well to illustrate the application of the proposed LSM.

Joint Modeling of Compensatory Multidimensional Item Responses and Response Times.

Computer-based testing (CBT) is becoming increasingly popular in assessing test-takers' latent abilities and making inferences regarding their cognitive processes. In addition to collecting item responses, an important benefit of using CBT is that response times (RTs) can also be recorded and used in subsequent analyses. To better understand the structural relations between multidimensional cognitive attributes and the working speed of test-takers, this research proposes a joint-modeling approach that integrates compensatory multidimensional latent traits and response speediness using item responses and RTs. The joint model is cast as a multilevel model in which the structural relation between working speed and accuracy are connected through their variance-covariance structures. The feasibility of this modeling approach is investigated via a Monte Carlo simulation study using a Bayesian estimation scheme. The results indicate that integrating RTs increased model parameter recovery and precision. In addition, Program of International Student Assessment (PISA) 2015 mathematics standard unit items are analyzed to further evaluate the feasibility of the approach to recover model parameters.

Bayesian DINA Modeling Incorporating Within-Item Characteristic Dependency.

The within-item characteristic dependency (WICD) means that dependencies exist among different types of item characteristics/parameters within an item. The potential WICD has been ignored by current modeling approaches and estimation algorithms for the deterministic inputs noisy "and" gate (DINA) model. To explicitly model WICD, this study proposed a modified Bayesian DINA modeling approach where a bivariate normal distribution was employed as a joint prior distribution for correlated item parameters. Simulation results indicated that the model parameters were well recovered and that explicitly modeling WICD improved model parameter estimation accuracy, precision, and efficiency. In addition, when potential item blocks existed, the proposed modeling approach still demonstrated good performance and high robustness. Furthermore, the fraction subtraction data were analyzed to illustrate the application and advantage of the proposed modeling approach.

Probabilistic-Input, Noisy Conjunctive Models for Cognitive Diagnosis.

Existing cognitive diagnosis models conceptualize attribute mastery status discretely as either mastery or non-mastery. This study proposes a different conceptualization of attribute mastery as a probabilistic concept, i.e., the probability of mastering a specific attribute for a person, and developing a probabilistic-input, noisy conjunctive (PINC) model, in which the probability of mastering an attribute for a person is a parameter to be estimated from data. And a higher-order version of the PINC model is used to consider the associations among attributes. The results of simulation studies revealed a good parameter recovery for the new models using the Bayesian method. The Examination for the Certificate of Proficiency in English (ECPE) data set was analyzed to illustrate the implications and applications of the proposed models. The results indicated that PINC models had better model-data fit, smaller item parameter estimates, and more refined estimates of attribute mastery.

Joint Testlet Cognitive Diagnosis Modeling for Paired Local Item Dependence in Response Times and Response Accuracy.

In joint models for item response times (RTs) and response accuracy (RA), local item dependence is composed of local RA dependence and local RT dependence. The two components are usually caused by the same common stimulus and emerge as pairs. Thus, the violation of local item independence in the joint models is called paired local item dependence. To address the issue of paired local item dependence while applying the joint cognitive diagnosis models (CDMs), this study proposed a joint testlet cognitive diagnosis modeling approach. The proposed approach is an extension of Zhan et al. (2017) and it incorporates two types of random testlet effect parameters (one for RA and the other for RTs) to account for paired local item dependence. The model parameters were estimated using the full Bayesian Markov chain Monte Carlo (MCMC) method. The 2015 PISA computer-based mathematics data were analyzed to demonstrate the application of the proposed model. Further, a brief simulation study was conducted to demonstrate the acceptable parameter recovery and the consequence of ignoring paired local item dependence.

Cognitive diagnosis modelling incorporating item response times.

To provide more refined diagnostic feedback with collateral information in item response times (RTs), this study proposed joint modelling of attributes and response speed using item responses and RTs simultaneously for cognitive diagnosis. For illustration, an extended deterministic input, noisy 'and' gate (DINA) model was proposed for joint modelling of responses and RTs. Model parameter estimation was explored using the Bayesian Markov chain Monte Carlo (MCMC) method. The PISA 2012 computer-based mathematics data were analysed first. These real data estimates were treated as true values in a subsequent simulation study. A follow-up simulation study with ideal testing conditions was conducted as well to further evaluate model parameter recovery. The results indicated that model parameters could be well recovered using the MCMC approach. Further, incorporating RTs into the DINA model would improve attribute and profile correct classification rates and result in more accurate and precise estimation of the model parameters.